Skip to content

docs(skill-evals): add a convention for recording live eval runs - #1626

Merged
potiuk merged 5 commits into
apache:mainfrom
thisisnic:docs/eval-run-log
Oct 19, 2026
Merged

docs(skill-evals): add a convention for recording live eval runs#1626
potiuk merged 5 commits into
apache:mainfrom
thisisnic:docs/eval-run-log

Conversation

@thisisnic

@thisisnic thisisnic commented Oct 13, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Summary

Design questions I'd like feedback on

  • Per-skill tables only, no cross-skill summary page. An earlier draft had a hand-maintained docs/eval-run-log.md listing all 81 suites; it was either a duplicate of the per-skill tables or a lossy digest, so it was dropped.
  • Refs are SHAs on main only; branch runs aren't logged. Simpler than tracking branch SHAs that rebases rewrite.
  • Model names are whatever the CLI calls them, no cross-provider normalisation.
  • Rows are append-only. No pruning rules yet; happy to add them if the tables get long.
  • Notes are links only (fix PR, issue, or a one-line reason for a grader flake). Anything longer belongs in the linked issue.

Type of change

  • Documentation (docs/, README.md, CONTRIBUTING.md)
  • Skill change
  • Tool / bridge contract
  • Python package
  • Groovy reference impl
  • Cross-cutting
  • Project template
  • CI / dev loop
  • Other:

Test plan

  • prek run passes on the changed files
  • prek run --all-files passes
  • N/A for evals: no skill or fixture text changes. The rows record runs made on main, not on this branch.

RFC-AI-0004 compliance

No principles touched — documentation only.

Linked issues

Generated-by: Claude Code (Fable 5.1)

🤖 Generated with Claude Code

@thisisnic

thisisnic commented Oct 13, 2026 •
edited
Loading

Copy link
Copy Markdown
Member Author

@potiuk - I was going to run some more of these but was after a bit of feedback on the design of this before I get into it more. I've kept it intentionally minimal for the moment, and was a bit hesitant to add a summary page as it'd need to keep in sync with the individuals suites, but alternatively, I could ditch the individual ones in favour of an overall summary page - what do you think?

@thisisnic
docs(skill-evals): add a convention for recording live eval runs
1c3babd 
CI never runs the eval suites against a live model, so the only record
of how a suite fares on a given model is whatever someone wrote down.
Nothing was written down. Add a "Recording runs" section to the harness
README defining a per-skill Run history table and four rules: main-only
SHAs as the ref, the model name as the CLI uses it, one row per batch of
identical runs, and record flakes.

Seed the first table in issue-reassess-stats with the pre-fix flake
observed on main, which apache#1615 addresses.

Generated-by: Claude Code (Fable 5.1)
@github-actions github-actions Bot added family:tools tools/* family:docs Docs, MISSION.md, READMEs family:issue issue-* skills capability:stats Read-only dashboards, metrics, governance evidence substrate:framework-dev Tool substrate: build / validate / eval the framework itself labels Oct 13, 2026
@potiuk

potiuk commented Oct 16, 2026

Copy link
Copy Markdown
Member

Yeah overall summary sounds good :)

This was referenced Oct 16, 2026
Merged
Closed
@thisisnic thisisnic mentioned this pull request Oct 17, 2026
Closed
@github-actions github-actions Bot added capability:triage Sweep + classify + propose disposition capability:fix Implement code change to resolve an issue capability:resolve Close-out: invalidate, dedupe, CVE allocate, announcement capability:reassess Re-run historical issues against current code labels Oct 17, 2026
@thisisnic
thisisnic marked this pull request as ready for review October 17, 2026 15:26
@thisisnic

thisisnic commented Oct 17, 2026

Copy link
Copy Markdown
Member Author

I ran these on some more evals in the "issues" group as well to get a feel for the kinds of things that come up often; this should be ready for review now.

potiuk
potiuk approved these changes Oct 19, 2026
View reviewed changes

@potiuk potiuk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A small, clear convention that fills a real gap — CI never runs a live model, so these tables are the only record of how a skill's evals actually behave across models. The seeded rows check out: both refs are on main, and every linked issue / PR is the right type (#1677 and #1615 merged fixes, #1678 and #1688 open). Thanks!

This review was drafted by an AI-assisted tool and
confirmed by an Apache Magpie maintainer. The maintainer
approving this PR has read the findings and signed off. If
something feels off, please reply on the PR and a maintainer
will follow up.

More on how Apache Magpie handles maintainer review:
Contributing guide.

@potiuk
potiuk merged commit 8234bcb into apache:main Oct 19, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@potiuk potiuk potiuk approved these changes

Assignees

No one assigned

Labels

capability:fix Implement code change to resolve an issue capability:reassess Re-run historical issues against current code capability:resolve Close-out: invalidate, dedupe, CVE allocate, announcement capability:stats Read-only dashboards, metrics, governance evidence capability:triage Sweep + classify + propose disposition family:docs Docs, MISSION.md, READMEs family:issue issue-* skills family:tools tools/* substrate:framework-dev Tool substrate: build / validate / eval the framework itself

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@thisisnic @potiuk